SYSTEM AND METHOD FOR OPTIMIZING SEARCH RESULT LISTINGS 

CROSS-REFERENCE TO RELATED APPLICATION 
This application claims the benefit of Provisional Application No. 60/535,353, filed 
5 January 9, 2004, which is hereby claimed under 35 U.S.C. § 119. 

FIELD OF THE INVENTION 
In general, the present invention relates to computer software and search engines and, 
in particular, to systems and methods for optimizing search result listings. 

BACKGROUND OF THE INVENTION 
10 The Internet search engine has become an important source of revenue for the service 

providers who operate them. The revenue is primarily generated from the display of 
advertisements to search engine users. The more Internet traffic that a search engine 
receives, the more attractive it is to advertisers, and the more revenue it can generate. It is 
generally regarded that the best way search engines can increase traffic is to provide highly 
15 relevant search results. But what is relevant today may not be relevant tomorrow, or even 
relevant later the same day. It is difficuh for service providers to keep pace with the rapid 
changes in searchable content based on seasonal and popular trends and topical events in the 
news. 

Complicating matters is the increasingly popular use of paid advertisements directly 
20 in the list of results that the search engine generates. The advertiser bids on popular search 
terms in exchange for which the search engine prominently lists their advertisement along 
with the other unpaid search results retumed for the bidded search term. For example, when 
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a user types in the search term "digital camera," the search results list might include a paid 
listing for Nikon® brand digital cameras preceding a relevant but unpaid listing for an 
independent digital photography Web site that reviews several brands of digital cameras. 

The practice of including paid listings along with the search results is commonly 

5 referred to as pay-per-click advertising, since the advertiser pays only when the user actually 
clicks on the listing (as opposed to more conventional Internet advertising, referred to as pay- 
per-impression, where the advertiser pays whenever the listing is displayed). Pay-per-click 
advertising is somewhat at odds with providing highly relevant search results, as the user 
might not consider the paid listings to be particularly relevant to their search. Cluttering up 

10 the search results with pay-per-click (PPC) listings in which users show little interest may 
impair the user experience to such an extent that traffic to the search engine site declines as 
users choose to conduct their Web searches elsewhere. Moreover, low-performing PPC 
listings, i.e., those with a low click-through rate, generate little revenue, regardless of how 
much the advertiser might have bid for the search term. Thus, the small revenue gained from 

15 low-performing listings may not be worth the risk of alienating users by displaying them 
along with the more relevant but unpaid search results. 

Since the amount of display area in a search results page is limited, some search 
engine operators place potentially less relevant search results as well as PPC listings for 
some or all bidded search terms off to one side-referred to as the sidebar-instead of placing 

20 them directly in line with the relevant unpaid listings. For instance, during the Thanksgiving 
season, the search term "turkey" might produce seasonal "in-line" listings for Thanksgiving 
turkey preparation (either unpaid or PPC listings) that would normally be placed in the 
sidebar with the twin goals of (a) increasing revenue for the search engine, and (b) increasing 
relevance of the results for the user. Search engine operators might also place potentially 

25 lower revenue paid listings for bidded search terms in the sidebar as well. Using the digital 
camera example, the search results list might include a PPC listing for competing Sony® 
brand- digital cameras placed in the sidebar next to the in-line listings for Nikon and the 
independent digital photography Web site, because the amount of Sony's bid for the search 
term "digital camera" was less than Nikon's. 
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In addition to maximizing the use of the available display area, sidebar placements 
have the advantage of being less distracting to the search engine user. However, experience 
has shown that the click-through rate for sidebar placements is inherently lower than the 
click-through rate for in-line placements, regardless of a particular listing^s click-through 

5 performance. Not surprisingly, users are simply less likely to notice and click listings 
appearing in a less desirable sidebar location than those appearing in the more desirable in- 
line location, i.e., directly in the list of search results. Consequently, one of the problems 
with placing listings in the sidebar is that such placements will likely generate less revenue 
and attract less repeat traffic than their in-line counterparts, even when the listings are 

10 otherwise high performers. Furthermore, as noted already, what is relevant today may not be 
relevant tomorrow, or even relevant later the same day. The determination of where to best 
place search result listings can change rapidly, making it difficult to place listings in the 
optimal location at any given time. 

SUMMARY OF THE INVENTION 

15 To overcome the above-described problems, a system, method, and computer- 

accessible medium for optimizing search result listings are provided. The system and 
method provide an automated way of promoting and demoting the placement of a search 
result listing between more or less desirable locations based on the listing's performance in a 
given placement. 

20 In accordance with one aspect of the present invention, the determination of when to 

promote or demote the placement of a listing to or from a given location is based on a 
threshold performance for the listing against which the actual performance of a listing is 
measured. The threshold performance for a listing may be fixed, or more preferably may be 
advantageously varied to accommodate different revenue and relevance goals for a particular 

25 search engine or portion of the search engine's market. 

In accordance with another aspect of the present invention, the system and method 
provide for an improved measurement of the performance of a listing by taking the listing's 
context into consideration. The listing's context may include a number of factors that are 
known to influence performance, including the location of the listing, the amount of display 

30 area that the listing occupies, the neighboring listings, and the display format characteristics 
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of the listing, e.g., the color, highlighting, animation, etc. When a listing's performance falls 
below the performance threshold, the listing may be demoted to a less prominent location or 
omitted from the search result listings altogether. Conversely, when a listing's performance 
exceeds the performance threshold, the listing may be retained in or promoted to a more 
5 prominent location. 

In accordance with a further aspect of the present invention, the search result listings 
may be promoted and demoted individually or as part of a set of results returned for a 
particular search term. When listings are promoted and demoted as part of a set of results, 
the system and method further provide for an improved measurement of the performance of a 
10 listing by taking the listing's position within the set of results into consideration. In addition, 
in accordance with a still further aspect of the present invention, the automatic promotion 
and demotion may be applied to any ranked search result, including both paid and unpaid 
listings. 

In accordance with yet other aspects of the present invention, a computer-accessible 
15 medium for optimizing search result listings is provided. The computer-accessible medium 
comprises data structures and computer-executable components comprising a listing 
promoter/demoter for automatically promoting and demoting search result listings based on 
actual performance measurements and performance thresholds. The data structures define 
search result listing and performance data in a manner that is generally consistent with the 
20 above-described method. Likewise, the computer-executable components are capable of 
performing actions generally consistent with the above-described method. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The foregoing aspects and many of the attendant advantages of this invention will 
become more readily appreciated as the same become better understood by reference to the 
25 following detailed description, when taken in conjunction with the accompanying drawings, 
wherein: 

FIGURE 1 is a depiction of an exemplary search result optimization system and one 
suitable operating environment in which search result listings may be optimized in 
accordance with the present invention; 
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FIGURE 2 is a block diagram depicting in further detail an arrangement of certain 
computing components of the search engine server of FIGURE 1 for implementing an 
embodiment of the present invention; 

FIGURE 3 is a pictorial diagram of an exemplary search engine user interface for 
5 implementing an embodiment of the present invention; 

FIGURE 4 is a pictorial diagram of the exemplary search engine user interface of 
FIGURE 3 at a later time, after the search result listings have been optimized in accordance 
with an embodiment of the present invention; 

FIGURE 5 is a block diagram of exemplary search result listings, their corresponding 
10 performance settings, and how the listings may be optimized in accordance with an 
embodiment of the present invention; 

FIGURE 6 is a block diagram of further exemplary search result listings, their 
corresponding performance settings, and how the listings may be optimized in accordance 
with an embodiment of the present invention; 
15 FIGURE 7 is a flow diagram illustrating the logic performed in conjunction with the 

search engine server of FIGURES 1 and 2 for promoting and demoting placements of search 
result listings in accordance with an embodiment of the present invention; 

FIGURE 8 is a flow diagram illustrating additional logic performed in conjunction 
with the search engine server of FIGURES 1 and 2 for measuring the performance of search 
20 result listings in accordance with an embodiment of the present invention; 

FIGURE 9 is a pictorial diagram illustrating an example implementation of entering a 
search term in an exemplary search engine user interface; 

FIGURE 10 is a pictorial diagram of an exemplary search engine user interface 
displaying the search result listings for the search term entered in FIGURE 9 in accordance 
25 with an embodiment of the present invention; and 

FIGURE 1 1 is a pictorial diagram of the exemplary search engine user interface of 
FIGURE 10 at a later time, after the search result listings for the search term entered in 
FIGURE 9 have been optimized in accordance with an embodiment of the present invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
The following discussion is intended to provide a brief, general description of a 
computing system suitable for implementing various features of an embodiment of the 
invention. While the computing system will be described in the general context of a personal 
5 and server computer or other types of computing devices usable in a distributed computing 
environment where complementary tasks are performed by remote computing devices linked 
together through a communication network, those skilled in the art will appreciate that the 
invention may be practiced with many other computer system configurations, including 
multiprocessor systems, minicomputers, mainframe computers, and the like. In addition to 

10 the more conventional computer systems described above, those skilled in the art will 
recognize that the invention may be practiced on other computing devices including laptop 
computers, tablet computers, personal digital assistants (PDAs), cellular telephones, and 
other devices upon which computer software or other digital content is installed. 

While aspects of the invention may be described in terms of programs or processes 

15 executed by a Web browser in conjunction with a personal computer or programs or 
processes executed by a search engine in conjunction with a server computer, those skilled in 
the art will recognize that those aspects also may be implemented in combination with other 
program modules. Generally, program modules include routines, subroutines, programs, 
processes, components, data structures, functions, interfaces, objects, etc., which perform 

20 particular tasks or implement particular abstract data types. 

FIGURE 1 is a depiction of an exemplary search result optimization system 100 and 
one suitable operating environment in which search result listings may be optimized in 
accordance with an embodiment of the present invention. As shown, the operating 
environment includes a search engine server 1 12 that is generally responsible for providing 

25 front-end user communication with various user devices, such as devices 102 and 104, and 
back-end searching services. The front-end communication provided by the search engine 
server 112 may include, among other services, generating text and/or graphics organized as a 
search Web page 106 using hypertext transfer protocols in response to information and 
search queries received from the various user devices, such as a computer system 102 and a 

30 personal digital assistant (PDA) 104. The back-end searching services provided by the 
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search engine server 112 may include, among other services, using the information and 
search queries received from the various user devices 102, 104 to search for relevant Web 
content, obtain commercial content, and track Web page and search result listing 
performance. 

5 In the environment shown in FIGURE 1, the search engine server 112 generates a 

search Web page 106 into which a user may input search terms 108 to initiate a search for 
Web content 120 via the Internet 118. The search terms 108 are transmitted to a search 
engine server 1 12 that uses the terms to perform a search for Web content 120 that is relevant 
to the search terms 108. The search engine server 112 relays the relevant Web content as a 

10 set of search results 1 10 for display to the user in the search Web page 106. 

In the environment shown in FIGURE 1, the user devices 102, 104 communicate with 
a search engine server 112 via one or more computer networks, such as the Internet 118. 
Protocols and components for communicating via the Internet are well known to those of 
ordinary skill in the art of computer network communications. Communication between user 

15 devices 102, 104 and the search engine server 112 may also be enabled by local wired or 
wireless computer network connections. The search engine server 112 depicted in 
FIGURE 1 may also operate in a distributed computing environment, which can comprise 
several computer systems that are interconnected via communication links, e.g., using one or 
more computer networks or direct connections. However, it will be appreciated by those of 

20 ordinary skill in the art that the server 1 12 could equally operate in a computer system having 
fewer or greater number of components than are illustrated in FIGURE 1 . Thus, the 
depiction of the operating environment in FIGURE 1 should be taken as exemplary and not 
limiting to the scope of the claims that follow. 

In one suitable implementation, the search optimization system 100 enables a search 

25 engine operator to advantageously optimize the placement of search results to maximize 
advertising revenue for the operator as well as relevance for the user. The search result 
optimization system 100 includes a search result optimizer process 122 that operates in 
conjunction with stored performance data 114 to optimize the search results 110 that are 
displayed in the search Web page 106. The stored performance data 114 includes the 

30 number of impressions of a particular search results listing, i.e., the number of times the 
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listing is displayed to the user on a search Web page 106 in response to the entry of a search 
term 108, as well as the number of clicks on the listing, i.e., the number of times a user clicks 
on the listing after it is displayed. The search engine server 112 is further configured to 
detect and filter out fraudulent clicks as is known in the art, such as spam clicking, simulated 
5 clicks by robots, and other suspect clicks such as multiple clicks from the same IP address 
within a certain amount of time or from unidentified sources. The performance of a 
particular listing is measured by the listing's click-through rate (CTR), which is determined 
by comparing the number of times the listing is displayed to the number of times the user 
clicks on the listing after it is displayed, i.e., dividing the number of impressions by the 

10 number of clicks. The stored performance data 1 14 may also include other data tracked by 
the search engine server 1 12, such as the location of the listing when it was displayed on the 
search Web page 106 and other characteristics of the listing that may influence performance, 
such as the color, size, font, animation, graphics, and adjacent listing performance data. 

The search terms 108 may include ordinary, unbidded terms 108 A on which 

15 advertisers have not bid, as well as bidded terms 108B on which advertisers have bid a dollar 
amount to have their commercial listing 116 displayed whenever the term is entered. In 
response to the search term entry, a search engine server 112 serves a user with search 
results 1 10 that the user can view via the search Web page 106. The search resuhs 110 may 
comprise both ordinary unpaid listings 11 OA that are obtained from the searchable Web 

20 content 120, as well as paid listings HOB that may be obtained from a commercial listings 
database 1 16. The paid listings 1 1 OB are primarily the type of pay-per-click (PPC) listings 
already described above, but could also include other types of commercial listings, such as 
paid directory listings and other sponsored listings assembled by the search engine operator. 
In operation, the search engine server 1 12 determines whether the search term 108 entered by 

25 the user is an ordinary, unbidded term 108 A, or a bidded term 108B. The search engine 
operator performs a search using the search term and, in addition, uses the bidded search 
term 108B to further obtain any corresponding commercial listings 1 16 that can be included 
in the search results 1 10. 

The unpaid listing 1 lOA, when clicked, links the user to Web content 120 relevant to 

30 the entered search term 108. When the paid listing 1 1 OB is clicked, it links the user to Web 
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content 120 provided by the advertiser corresponding to the paid listing HOB. As the user 
clicks on the listings 11 OA and HOB that comprise the search results 110 displayed on the 
search Web page 106, the search engine server 112 captures the performance data 114 for 
each listing, including data that may aid in interpreting the performance of the listing, such as 

5 the context of the listing, i.e., the location of the listing on the Web page 106, the amount of 
display area that the listing occupies, the neighboring listings, and the display characteristics 
of the listing, e.g., the color, highlighting, animation, etc. From the performance data 114, 
the search optimization system 100 is able to derive and interpret certain statistical 
information about the listing, particularly the above-described CTR. 

10 FIGURE 2 is a block diagram depicting in further detail an arrangement of certain 

exemplary computing components of the search engine server 112 that are responsible for 
the operation of the search result optimization system 100 shown in FIGURE 1. Specifically, 
the search engine server 1 12 is shown including an operating system 202, processor 203, and 
memory 206 to implement executable program instructions for the general administration 

15 and operation of the search engine server 1 12. The search engine server 112 further includes 
a network interface 204 to communicate with a network, such as the Internet 1 18, to respond 
to user search terms 108 and to provide search results 110. Suitable implementations for the 
operating system 202, processor 203, memory 206, and network interface 204 are known or 
commercially available, and are readily implemented by persons having ordinary skill in the 

20 art — particularly in light of the disclosure herein. 

The memory 206 of the search engine server 112 includes computer executable 
program instructions comprising the search resuh optimizer process 122. In some 
embodiments, the memory 206 may further include various stored data, such as the above- 
described search terms 108 and search results 1 10, as well as performance settings 208. The 

25 search result optimizer process 122 uses the performance settings 208 against which to 
measure the performance of a search result listing and to place the listing in an optimal 
location on the search Web page 106, as will be described in further detail below. 

The search result optimizer process 122 includes a listing promoter/demoter 
process 123 and a performance measurement process 124. The listing promoter/demoter 

30 process 123 operates in conjunction with the performance measurement process 124 to 
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promote and demote one or more listings in the search results 110 from a more prominent 
location on the search Web page 106 to a less prominent location, and vice versa, based on a 
measurement of the listing's performance in a given location. The more prominent location 
on the search Web page 106 is generally the location that is in line with the placement of the 
5 main search result listings, and the less prominent location is generally the location that is off 
to one side of the search result listings, such as the sidebar. Listings with better performance 
are generally maintained in or promoted to the more prominent location on the search Web 
page 106, whereas listings with poorer performance are generally maintained in or demoted 
to the less prominent location. 

10 The performance measurement process 124 measures the performance of a listing 

using the listing's CTR, as well as the listing's context as obtained from the stored 
performance data 114 that was previously captured by the search engine server 112, as 
described above. The measurement of the performance is based on stored performance 
settings 208. The stored performance settings 208 are tunable settings that include certain 

15 threshold performance levels against which the performance measurement process 124 
measures the performance of a listing in a given location. For example, the threshold 
performance level for a listing in a more prominent location on the search Web page 106 
may be a 5% CTR, while the level for a listing in a less prominent location may be only a 
3% CTR. Listings in the more prominent location, but with CTRs that fall below 5%, may 

20 be demoted to a less prominent location on the search Web page 106, whereas listings in the 
less prominent location, but with CTRs that meet or exceed 3%, may be promoted to the 
more prominent location. The settings 208 reflect, at least in part, the search engine 
operator's expectations for listing performance in a given location, and may be tuned to 
accommodate different expectations of performance in one search engine market to the next 

25 to further the search engine operator's goals for advertising revenue and search results 
relevance. 

Since the location of a listing on the search Web page 106 significantly influences the 
CTR of the listing, it is important that the decision to promote or demote a listing is based on 
accurate measurements of performance. Accordingly, in a preferred embodiment, the 
30 performance measurement process 124 takes into account the listing's context as it appeared 
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in the search Web page 106 when measuring the performance of the Usting, including the 
current location, size, color, and any other characteristic of the listing that is known to affect 
the CTR. For example, placing a listing in a more prominent location alone may raise the 
CTR by at least one percentage point. Thus a listing that has a CTR of 4% in a more 
5 prominent location is performing about the same as if it had a CTR of 3% in the less 
prominent location. Such a performance might not justify keeping the listing in the more 
prominent location, but could justify keeping the listing in the less prominent location. 

To further improve the accuracy of the measurement of performance, in one 
embodiment, the search result optimizer 122 may employ statistical algorithms in the listing 

10 promoter/demoter 123 and performance measurement 124 processes to ensure that only 
statistically significant measurements of actual performance and comparisons to expected 
performance from one point in time to the next form the basis for making a decision to 
promote or demote a given listing or set of listings. 

FIGURE 3 illustrates a browser program 300 displaying a Web page 106 in which is 

15 depicted an exemplary search engine user interface for implementing an embodiment of the 
present invention. The Web page 106 may be generated by the search engine server 112 and 
delivered to the user's computing device 102, 104 via the Intemet 118. The search engine 
user interface displays the previously entered search terms 108 in the text box 302 and 
prompts the user to refine the search with additional search terms, if desired, using the 

20 command button labeled "REFINE SEARCH" 304. The search engine user interface 
displays the search results 110 in one of two or more locations on the Web page 106 in 
FIGURE 3, including a more prominent in-line result location 306 and a less prominent 
sidebar result location 308. In the illustrated example, the Web page 106, here shown at time 
9:00 a.m., as depicted in time box 324, includes an in-line result location 306 that is typically 

25 used to display the relevant search results obtained for the search term, displaying 
Listing A 310, Listing B 312, Listing C 314, as well as any paid listings that qualify for 
inclusion in the in-line location 306 based on their performance. In one embodiment, the 
paid listings may qualify for inclusion in the in-line location 306 only when their CTR is at 
or above the minimum threshold performance setting for the in-line location 306. The less 

30 prominent location, here the sidebar resuhs location 308 is typically used to display less 
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relevant listings or paid listings with as yet unproven performance — here displaying 
Listing X 318, Listing Y 320, and Listing Z 322. As with the in-line location, listings may 
qualify for inclusion in the sidebar location 308 only when their CTR is at or above the 
minimum threshold performance setting for the sidebar location 308. The search engine user 
5 interface may include other hypertext links, such as a "Next" link 326, providing a link to 
additional Web pages not illustrated. The Next link 326 may produce, for example, 
additional search results containing the search term listed in box 302. 

FIGURE 4 is a pictorial diagram of the exemplary search engine user interface of 
FIGURE 3, but at a later time — here 10:00 a.m., as shown in time box 324 — after the search 

10 results Listings A-Z provided in response to the search term entered in box 302 have been 
optimized in accordance with an embodiment of the present invention. In the illustrated 
example, Listing Z 322 apparently outperformed its former location in the sidebar 
results 308, whereas Listing A 310 apparently underperformed its former location in the in- 
line results 306. Consequently, the in-line results location 306 has been altered to demote 

15 Listing A 310 from its in-line location in FIGURE 3, and to promote Listing Z 322 from its 
sidebar location in FIGURE 3 to its new location in FIGURE 4 in the in-line results 306. 
Conversely, the sidebar results location 308 has been altered to promote Listing Z 322, and 
to add the demoted Listing A 310 that was formerly located in the in-line results 
location 306. 

20 FIGURES is a block diagram depicting exemplary search result listings, their 

corresponding performance settings, and how the placement of the listings may be optimized 
in accordance with an embodiment of the present invention. In the example, an expected 
CTR 502 for Listings X, Y, and Z may be set to respective percentages 4%, 2%, and 1%, i.e., 
4 click-throughs for every 100 impressions, 2 click-throughs etc. When the listings are 

25 located in a more prominent in-line location, such as the in-line results 306 previously 
illustrated in FIGURES 3-4, the expected CTR, as adjusted for the in-line location, may be 
respectively, 5%, 3%, and 2%. When the listings are located in a less prominent location 
such as the sidebar results 308 previously illustrated in FIGURES 3-4, the expected CTR, as 
adjusted for the sidebar location, may be respectively 3%, 1%, and .5%. 
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In operation, the search engine server 112 tracks the actual performance of the 
Listings X, Y, and Z at 508, resulting in actual performances that may be respectively 4%, 
3%, and 2%. When the Listings X, Y, and Z are located in the more prominent in-line 
location, the performance measurement process 124 of the search result optimizer 
5 process 122 measures the actual performances 508 against the in-line adjusted expected 
CTR 504. When the Listings X, Y, and Z are located in the less prominent in-line location, 
the performance measurement process 124 of the search result optimizer process 122 
measures the actual performances 508 against the sidebar adjusted expected CTR 506. 
Consequently, when the Listings X, Y, andZ are located in the more prominent in-line 

10 location, the performance measurement process 124 determines that the in-line 
performances 510 of the listings are such that Listing X, at 4%, is underperforming the in- 
line expected CTR 504, while Listings Y and Z, at 3% and 2%, respectively, are performing 
just at the in-line expected CTR 504. On the other hand, when the Listings X, Y, and Z are 
located in the less prominent sidebar location, the performance measurement process 124 

15 determines that the sidebar performances 512 of the listings are such that Listing X at 4% is 
outperforming the sidebar expected CTR 506, as are Listings Y and Z. Based on those 
determinations of the performance measurement process 124, the listing promoter/demoter 
process 123 would demote Listing X and retain Listings Y andZ based on their in-line 
performance 510, but would promote Listings X, Y, andZ based on their sidebar 

20 performance 512. 

In some cases, the search result optimization system 100 may optimize the placement 
of listings on a set-by-set basis instead of, or in addition to, a listing-by-listing basis. 
However, depending on the performance of the listings that comprise a given set, this might 
lead to less than optimal placements of listings. To address this possibility, the search result 

25 optimizer process 122 may be further configured to tier the search result listings to take 
certain characteristics of a set into account when measuring performance, such as the number 
and position of listings in the set, or the size of the listings themselves. 

FIGURE 6 is a block diagram of further exemplary search result listings, their 
corresponding performance settings, and how the listings may be optimized in accordance 

30 with an embodiment of the present invention. In the illustrated example, Listings X, Y, 
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and Z comprise a tiered result set 602. A tiered result set 602 is a set of search result listings 
for which there is an overall expected CTR for the set that depends on the number and/or size 
of results in a set. This is because the greater the number of listings that are returned in a 
given search results set, or the larger the listings are themselves in the search results set, the 

5 larger the amount of space the search results 110 (in FIGURE 1) will occupy on the Web 
page 106 (in FIGURE 1). For this reason, the expected CTR for larger result sets is 
generally higher than the expected CTR for smaller result sets. Moreover, the expected CTR 
for any individual listing may vary based on its position within the set. Thus, the first listing 
in a set may have a higher expected CTR than the listings appearing further down in the set 

10 of results. Without tiering, the actual CTR for a set would be based on an average CTR of 
the individual listings of the set. Averaging the CTR could resuh in a set having listings with 
mediocre performance that average up to the minimum overall CTR for the set, leading to an 
undesirable promotion to or retention of the set in a particular location. 

Using tiering, the performance measurement process 124 may compare the overall 

15 performance of a set of listings to an expected CTR for the set that has been adjusted to take 
into account the number and size of the listings in the set. For instance, in the illustrated 
example at block 604, the expected CTR for the individual Listings X, Y, and Z of a result 
set is 2%, 2%, and 1%, respectively, resulting in an average CTR for the set of 1.7%. As 
illustrated in block 606, since Listing X is in the top position, the expected CTR for 

20 Listing X using tiering is upwardly adjusted to 3% from 2%. Thus, the new average CTR for 
the tiered set is (3+2+1) / 3 = 2%. In addition, the new average CTR for the tiered set is also 
upwardly adjusted to an overall CTR of 3% from 2% to reflect the number and size of the 
Listings X, Y, and Z in the set. 

In operation, the search engine server 1 12 captures performance data 1 14 that reveals 

25 that the actual performance 608 of the individual Listings X, Y, and Z returned in result 
set 602 is 2% each, resulting in an average CTR for the set of 2%. As illustrated in 
block 6 10, were the performance measurement process 124 to measure the performance of 
the set 602 without using tiering, the set would be considered to have exceeded the nontiered 
average expected CTR of 1.7% at 604. Moreover, the set would have even met the tiered 

30 average expected CTR of 2% at 606. Either way, the set would have qualified for retention 
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in or promotion to a more prominent location on the Web page 106, even though the 
individual performance of the top Listing X was a mediocre 2%, well below the upwardly 
adjusted expected CTR for Listing X of 3% at 606 based on its position at the top of the set. 
But, as illustrated in block 612, using tiering, the performance measurement 

5 process 124 instead measures the performance of the set 602 against the tiered adjusted 
CTRs for the set and the listings. Thus, in the illustrated example, the performance 
measurement process 124 determines that the actual performance 608 of the set of 2% is less 
than the tiered adjusted CTR of 3% for the set. In addition, the performance measurement 
process 124 determines that the actual performance of Listing X of 2% is less than the tiered 

10 adjusted CTR of 3% for the listing. Accordingly, neither the set nor the listing qualifies for 
retention in or promotion to a more prominent location on the Web page 106, and would 
instead be demoted or removed from the Web page 106 altogether. 

FIGURE 7 is a flow diagram illustrating the logic performed in conjunction with the 
search engine server of FIGURES 1 and 2 for promoting and demoting placements of search 

15 result listings in accordance with an embodiment of the present invention. The optimizer 
process 122 begins at the start block 702 and continues at processing block 704 to obtain the 
actual performance data of the listings from the stored performance data 114. As described 
above, the actual performance data may include not only the impression and click data, but 
may also include other data known to influence the performance of a listing, such as the 

20 location, the position of a listing within a set, the size of the listing, i.e., how much space the 
listing occupies on the Web page 106, and any display format characteristics that enhance or 
otherwise alter the appearance of the listing, such as the color, animation, graphics, 
highlighting, etc. 

At processing block 706, the optimizer process 122 obtains the performance 
25 settings 208 against which the performance of a listing or a set of listings is measured. The 
performance settings 208, as already described above, are tunable settings that may vary 
from one implementation of the search result optimization system 100 to the next, and may 
even be changed during the operation of the search result optimization system 100 to meet 
the revenue and relevance goals of the search engine operator. The performance settings 208 
30 may include expected performance levels for listings and/or sets of listings. In one 
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embodiment, the performance settings 208 may include expected performance levels based 
on location, size, or other data known to influence the performance of a listing. 

At processing block 708, the optimizer process 122 measures the actual performance 
of a listing or set of listings against the expected performance by performing the performance 
5 measurement process 124, the processing logic of which is explained in further detail with 
reference to FIGURE 8 below. Once the measurement has been completed, the optimizer 
process 122 continues at decision block 710 to determine whether the listing (or set of 
listings) should be promoted or demoted. The decision will depend on the performance 
measurement as determined in performance measurement process 124 and the current 

10 placement or location of the listing or set of listings under consideration. For example, when 
the listing or set of listings is in a less prominent location but has exceeded the performance 
levels for that location, then the optimizer process 122 may promote the listing. On the other 
hand, when the listing or set of listings is in a more prominent location but has failed to meet 
the performance levels for that location, then the optimizer process 122 may demote the 

15 listing. These are just two examples of decisions that the listing promoter/demoter may 
implement; other variations on the decision-making process may be implemented at decision 
block 710, such as the decision to retain a listing or set of listings in its current location, 
without departing from the scope of the claims that follow. 

Once the decision to promote or demote a listing or set of listings has been made, the 

20 optimizer process 122 continues at processing block 712 to implement the decision by 
configuring the listings according to the promotion/demotion decision. For example, in one 
embodiment, the optimizer process 122 may configure the commercial listings database 1 16 
to configure a paid listing that has been demoted to appear in a less prominent location on the 
Web page 106 when it is next included in a set of search results 110 by search engine 1 12, 

25 and vice versa. In another embodiment, the optimizer process 122 may configure the search 
results 1 10 in real time to place a listing that has been demoted in a less prominent location 
when it is displayed on the Web page 106, and vice versa. After the configuration 
process 712 is completed, the optimizer process 122 logic ends at termination block 714. 

FIGURE 8 is a flow diagram illustrating additional logic performed in conjunction 

30 with the search engine server 1 12 of FIGURES 1 and 2 for measuring the performance of 
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search result listings in accordance with an embodiment of the present invention. In one 
embodiment, the performance measurement process 124 begins at the start block 802, and 
continues at processing block 804 to normalize the actual performance data for the listing or 
set of listings under consideration to reflect the current context of the listing or set of listings, 
5 e.g. the location, position of listing within a set, etc. Normalization of the actual 
performance data includes adjusting the data upwards to compensate for contextual factors 
that are known to negatively affect the CTR and, conversely, adjusting the data downward to 
compensate for contextual factors that are known to positively affect the CTR. The 
contextual factors that are considered include location, position, appearance, etc., and any 

10 other factor known to influence the CTR. At processing block 806, the performance 
measurement process 124 continues at processing block 806 to compare the adjusted 
performance of the listing or set of listings to the expected performance settings as 
previously obtained from the stored performance settings 208. 

In another embodiment, the adjustments performed in process block 804 may not be 

15 necessary when the adjustments in the expectations for performance have been reflected in 
the stored performance settings themselves. In other words, the listing's context may be 
taken into account by incorporating the reduced or enhanced expectations into the 
performance settings against which the actual performance of a listing is measured or, 
altematively, the listing's context may be taken into account programmatically during the 

20 performance measurement process. This is the case, for example, when the performance 
settings are configured to include one expectation for performance when a listing is placed in 
a more prominent location, and another (reduced) expectation for performance when the 
listing is placed in a less prominent location. Moreover, combinations of the foregoing 
alternatives may be employed without departing from the scope of the claims that follow. 

25 The performance measurement process 124 continues at decision block 808 to 

determine whether the actual performance exceeds the expected performance after taking the 
context of the listing into account. If so, the performance measurement process 124 flags the 
listing or set of listings under consideration for promotion at preparatory block 810. If not, 
the performance measurement process 124 flags the listing or set of listings under 

30 consideration for demotion at preparatory block 810. The performance measurement 
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process 124 returns at termination block 814 to the optimizer process 122, described above 
with reference to FIGURE 7. 

FIGURE 9 illustrates a browser program 900 displaying a search Web page in which 
the user is able to enter one or more search terms in a search entry box 902. The Web page 

5 may be generated by the search engine server 112 and delivered to the user's computing 
device 102, 104 via the Internet. The user enters the desired search terms in the box 902 and 
uses a pointing device, keyboard, or other input device to initiate the search, e.g., by clicking 
on the button "SEARCH" identified at reference numeral 904. In the illustrated example in 
FIGURE 9, the user has entered the search term "DOG FOOD." The results of the user's 

10 search are displayed to the user, as illustrated in FIGURE 10. 

FIGURE 10 illustrates the browser program 900 displaying a Web page generated at 
1:00 p.m., as displayed at time box 1016, and in which listings comprising the search results 
for a search term "DOG FOOD" are displayed. The search results are generated from a 
search of Web content 120 via the Internet 118, as well as from a commercial listing 

15 database 116 operated by the search engine operator. As is known in the art, the search 
engine 112 may execute program instructions that analyze the listings and rank the "best" 
listings for display to the user according to a predetermined criterion, such as which pages 
are most relevant. The best listings are generally displayed in more prominent locations, 
such as the in-line location previously described, or at the top of a set of listings. The search 

20 results may be displayed in a variety of different formats and in different locations on the 
Web page. In one embodiment of the present invention, the search result optimization 
system 100 optimizes the placement and display of the listings according to the listing 
performance and in furtherance of the search engine operator's goals for revenue and 
relevance, as described with reference to FIGURES 7 and 8. 

25 By way of example only, directly beneath the redisplayed search entry box 1002 and 

to the left of the Web page, three listings are displayed in accordance with their performance 
as measured at 1:00 p.m., as shown in time box 1016 — (1) PETsMART® listing at reference 
numeral 1006; (2)eBAY® listing at reference numeral 1008; and (3) Wild Alaska Salmon 
Dog Food Supplement listing at reference numeral 1010. These three listings occupy the 

30 more prominent location on the Web page. The PETsMART and eBAY listings are likely to 
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be paid listings that are listed not only because they are relevant, but also because they are 
bidded listings associated with bids on the search term "DOG FOOD." Off to the right side 
of the Web page, across from the first three listings, is one additional listing for the book 
"Dog Food" by author Saxton Freymann, from Internet bookseller Amazon.com at reference 
5 numeral 1012. The Amazon.com listing may also be a paid listing, but occupies the less 
prominent location on the Web page, referred to as the sidebar, because for this optimization 
of search results for search term "DOG FOOD," it may be presumed to be less relevant," 
having to do v^th a book that just happens to be titled "DOG FOOD" as opposed to dog food 
itself Again, by way of example only, the PETsMART listing at reference numeral 1006 is 

10 animated with a graphic for a pet food calculator, as is the Amazon.com listing at reference 
numeral 1012, which is animated with the book cover of the "Dog Food" book. The 
graphical animation is just one example of the listing context that may influence (in this case 
enhance) the performance of the listing. 

FIGURE 1 1 illustrates the browser program 900 displaying another Web page, this 

15 time generated later at 3:00 p.m., as displayed at time box 1016, and again in which listings 
comprising the search results for a search term "DOG FOOD" are displayed. In the 
illustrated example, it appears that the performance for the paid listing for PETsMART at 
reference numeral 1006 did not justify retaining the listing in the more prominent location 
that it occupied earlier, as shown in FIGURE 10. For example, if the performance 

20 expectation for the listing was 4% CTR, adjusted to 5% CTR because of its position at the 
top of the set, or because of the pet food calculator graphic enhancement, and the actual 
performance in the last two hours was only 2%, then the listing would be a candidate for 
demotion to the position shown in FIGURE 1 1, in the sidebar location to the right of the Web 
page. 

25 Furthermore, in the illustrated example, it is apparent that the performance for the 

Amazon.com listing at reference numeral 1012 for the book "Dog Food" is exceeding the 
expected performance sufficiently, since the last optimization at 1:00 p.m. justifies 
promoting the listing to the more prominent in-line location directly beneath the search term 
entry box 1002 as shown in FIGURE 11. Circumstances that may have caused such a 

30 dramatic change in the performance of the listing might be, for example, that the book "Dog 
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Food" was featured on a television show or in a book review, or that the Christmas buying 
season has kicked in and the book is a hot Christmas gift item. This boost in performance is 
probably temporary, but optimizing the search results listings to respond quickly to the 
changed circumstances for the search term "DOG FOOD" could have a tremendous impact 
5 on revenue earned from the Amazon.com listing, not to mention the relevance for the user, 
and the anticipated increase in return traffic to the search engine operator's Web page that 
satisfied users will generate. 

Other example optimizations illustrated in FIGURE 1 1 include the promotion of the 
listing for PETCO at reference numeral 1 102 to be included in the first page in-line results. 

10 Notice that the listing for PETCO is below the eBAY listing at reference numeral 1008, 
which remains unchanged. This may be because the amount bid by eBAY for inclusion in 
the search results set for the search term "DOG FOOD" is higher than that bid by PETCO, 
and the performance of the listing was sufficient to retain the listing in the same in-line 
location as it earlier appeared. Using the performance optimization process 100, however, 

15 the order of the listings for eBAY and PETCO may be switched should the performance of 
the PETCO listing continue to exceed expectations, and the performance of the eBAY listing 
fall short of expectations. 

While the presently preferred embodiments of the invention have been illustrated and 
described, it will be appreciated that various changes may be made therein without departing 

20 from the spirit and scope of the invention. For example, in one embodiment of the present 
invention, the search optimization system 100 processes may be implemented in real time to 
allow for up-to-the-minute optimizations based on the latest performance data captured by 
the search engine server 112. In another embodiment, the search optimization system 100 
processes may be implemented in batch mode to allow for a combination of automated and 

25 manual optimizations to the configuration of search result listings. In yet other 
embodiments, the search result optimization system may be limited in application to paid 
listings or unpaid listings, ranked or unranked listings, or any combination thereof 
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